Augmentations based on positioning accuracy or confidence

ABSTRACT

Augmented reality augmentations are selected or modified based on accuracy or confidence information concerning locations and orientations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/512,282, filed May 30, 2017, the complete contents ofwhich are herein incorporated by reference.

FIELD OF THE INVENTION

The invention generally relates to virtual reality and augmented realityrepresentations and, in particular, controlling representations based onaccuracy or confidence measures of location and orientation information.

BACKGROUND

In augmented reality systems, augmentations are output to a user toalter their experience of a real world environment. Often an augmentedreality (AR) system will superimpose images such as shapes or icons ontop of a real world view captured by a camera or else seen directlythrough a see-through head mounted display (HMD).

Sports broadcasters frequently use augmented reality to representinformation to viewers which is visible in the real world. For instancewith American football, the broadcasting company frequently augments therecorded image of the real world view with the line of scrimmage andfirst down markers on the field. The line and markers do not exist inreality, but rather they are virtual augmentations that are added to thereal world view. In order to accurately place the augmentations withrespect to the real world view, the cameras and systems for recordingthe gameplay are set up with what may be referred to as “bore siting,”which involves careful alignment of cameras with the playing field sothe locations of both the camera and the field are known within a highdegree of accuracy (high confidence). This helps ensure that the firstdown markers, for instance, are not shown 10 feet away from where theyactually belong.

To “augment” the “reality” of the real world in a convincing andrealistic manner, there should be some relationship between theaugmentation and objects which are visible in the real world image. Ingeneral, an AR system must have some understanding of location of the ARsystem's camera and the objects within view of the camera in order toprovide an augmentation to the user which has a meaningful relationshipwith the user's real world surroundings.

SUMMARY

If, for example, an AR system is configured to help users find aspecific building and is displaying an augmentation intended to markthat building in a live video stream viewed by the user as he or shewalks down the street, it is important to either exactly mark thecorrect building, or to otherwise indicate to the user that there issome uncertainty in the system's locationing accuracy. A system thatmerely takes its best guess and then marks the wrong building, with noindication to the user of a potential accuracy risk, could send the userunwittingly to the wrong location. The user would perhaps see an arrowaugmentation, sized to be much smaller than the building itself,pointing at the center of the wrong building. Such is a type of problemor error addressed by exemplary embodiments of the invention.

According to an aspect of some exemplary embodiments, augmented realityaugmentations are provided to a user in such a manner that they conveyan accuracy or confidence measure of the location and/or orientationinformation which was used to determine the spatial relationship betweenthe user and an object or the spatial relationship between two objects.

According to another aspect of some exemplary embodiments, accuracy orconfidence measures of location and/or orientation information areconveyed to users via alternative appearances of virtual objects whichrelated to the location and/or orientation information.

As an illustrative example, the aim of a particular augmented realityalgorithm may be to display a label to a user which identifies a realworld structure as matching a particular address or business name whichmay not itself be apparent from the building's exterior. Theaugmentation may generally take the appearance of a callout (e.g., aspeech bubble) which, under ideal circumstances, points to or emanatesfrom the front door of the building. To achieve this objective, an ARsystem or device must know with great accuracy and precision thelocation of the building, even more specifically the location of thefront door, the user's location, and the user's orientation with respectto the building and front door. Is the user 10 feet from the door or 200feet from the door? Is the user directly facing the door from across thestreet, or is the user behind the building looking in the direction ofthe front door but unable to see it because the building's walls obscurethe user's view of the door? Is the door at the same elevation as theuser or the building, or is it positioned up or down a flight of stairsand therefore at some different elevation? These and other positioningand orientation considerations all affect where the callout augmentationshould be superimposed on the real world image. If the door is furtheraway, perhaps the callout should be smaller. If the door is closer,perhaps the callout should be larger. If the door is to the left or theright of the real world image, then the bubble should also be more tothe left or more to the right of the image, respectively. A problem,however, is that this level of accuracy or precision of location may notbe available to the AR system. For instance, the location of thebuilding may only be known within a margin of error of 10 meters. Or, itmay be known that the door of the building is on the north facing wall,but it is unknown whether the door is at the left end of the north wall,the right end of the north wall, or somewhere in between, which mayconstitute a difference of few hundred meters or more depending on thesize of the building.

An exemplary embodiment according to this disclosure attends to theproblem described in the preceding paragraph by selecting or changingthe augmentation (e.g., the appearance of the virtual object, in thiscase the callout) based on the extent of the accuracy and/or precisionof the locations of the building, door, and user. For instance, if thebuilding's location is only known within a margin of error of 100meters, and as a result any of three different buildings might be the“right building” over which to superimpose the callout, the calloutmight be made large enough to stretch over all three buildings and pointto none of them. Alternatively, the callout could point one of the threebuildings and be color coded to signify to a user that the bubble may beas much as 100 meters off from the actual location it belongs.Alternatively, the bubble may be shown with deliberately poor sharpness(high blur) to signify to a user that the accuracy of its placement isin doubt or that its placement has a certain margin of error. On theother hand, the more accurate the known location of the building, thebetter the sharpness of the callout would be made by the AR device. Avariety of different modifications or characteristics may be made to theaugmentation or selected prior to the display or output of theaugmentation depending on the accuracy or precision of object locationsknown to the AR device.

Characteristics of augmentations which may be selected or altered basedon accuracy of positioning include but are not limited to blur/sharpness(e.g., fuzzier versus clearer lines), size/extent (e.g., bigger orsmaller), color, and shape (e.g., point versus a cloud). Positioningaccuracy may also or alternatively be conveyed by a base icon that doesnot necessarily change but which is accompanied (e.g., bracketed) byuncertainty indicators. For instance, a point of constant size may bebracketed by an uncertainty indicator that indicates positioninguncertainty based on the extent to which the indicator extends outwardfrom the base icon. The extent to which the indicator extends outsidethe base icon indicates the area of uncertainty. For instance, a circleof variable size may be used for the indicator, the circle extending outaround a point or dot of constant size.

A constellation of visualization may be provided for the same visualtarget (e.g., the building or door in the example of the precedingparagraph), and the node of the constellation is chosen or switchedbased on the real time changes of the estimated accuracy of positioning.In essence, each node corresponds with a different augmentation, and adifferent augmentation is output based on different accuracy measures.For visual augmentations, each node may correspond with a differentgraphical artifact.

Many exemplary embodiments concern visual augmentations, but auditoryand tactile augmentations are also suited for the same principlesdiscussed herein. Using the building and door example above, if theaccuracy of building location is not well known, an auditoryaugmentation may be provided which explains “The building is near,”whereas if the accuracy of the building location is well known, anauditory augmentation may be provided which explains “The building isdirectly in front of you.”

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary method for providing an augmented reality.

FIG. 2 is an exemplary system for providing an augmented reality.

FIGS. 3A and 3B show an exemplary augmented reality device.

DETAILED DESCRIPTION

Augmented reality involves defining spatial relationships betweenvirtual objects and real objects, and then making the virtual objectsapparent to a user of the augmented reality system in such a way as tocombine real and virtual objects. For example, a visual augmentedreality display could use virtual and real objects, and their definedspatial relationships, to generate a combined visual display in the formof a live streaming video (presenting real objects) overlaid withrepresentations of the virtual objects. A spatial relationship betweentwo objects (either or both of which may be virtual or real) may involveone or more of a topological relation, a distance relation, and adirectional relation. A topological relation between an object A and anobject B may be, for example, A is within B, A is touching B, A iscrossing B, A is overlapping B, or A is adjacent to B. Precise spatialrelationships between real and virtual objects allow an augmentedreality system to generate perceptual experiences in which real andvirtual objects are apparently combined seamlessly, e.g. for visualsystems the combined presentation is apparently in the correct visualproportions, perspectives, and arrangement. Without correct reckoning ofthe spatial relationships in such a system, errors in the presentationof the system's output to the user can cause the system to be unusable,e.g. virtual objects appear out of place and therefore are not useful.An example is a virtual visual label that should label one building, butis erroneously shown overlaid onto a different building.

In order to create a visual augmented reality system, in addition toestablishing spatial relationships between virtual objects and realobjects, the visual perspective into the real world must be matched tothe effective visual perspective into the virtual world. Even when thevirtual world objects are sized and positioned correctly with respect totheir real world counterparts, the determination of which virtualobjects are eligible for visual presentation to the user depends on theperspective in the virtual world, which must be matched to the realworld perspective of a real world camera in order to take advantage ofcarefully determined spatial relationships among virtual and realobjects. For instance, it may be insufficient to know whether a man withan AR headset is two feet away from a building. It is also necessary toknow whether the man is facing toward the building or, alternatively,facing away from the building. This involves knowing the orientation ofthe camera. The pose of a camera includes the position and orientationof the camera. The perspective of the camera includes the position ofthe camera, the orientation of the camera, and the field of view of thecamera.

Location information for a camera or an AR device may be obtained, forexample, from a GPS system co-located with the camera or AR device. AnAR device may, for example, have a GPS unit built-in at the time ofmanufacturing. Location accuracy or confidence information may beobtained from the same source or sources as location information.Existing GPS units on the market provide, in addition to latitude,longitude, and altitude information, accuracy measures for each of thesethree measures. GPS units may give both a horizontal accuracy estimateand vertical accuracy estimate. The accuracy measures may be referred toby other names such as confidence estimates or margins of error. Theactual value of the accuracy may depend on a number of circumstanceswhich may include but are not limited to: the accuracy with which thecamera is mounted with respect to the body of the AR device, and theaccuracy of referential location information stored in a database. Insome instances the accuracy may not be known in which case it may beestimated or assumed, in which case the selected augmentation is basedon the estimated or assumed accuracy. Moreover, exemplary embodimentsmay be configured to accommodate many different kinds of confidenceintervals. This is advantageous since not all devices (e.g., location ororientation sensors) necessarily represent accuracy, certainty, orconfidence in the same way or form.

Orientation information may be obtained from sensors like a gyroscopeand digital compass. These orientation devices, like GPS units or otherlocation determining units, may be configured or configurable to provideaccuracy or confidence measures in additional to their main outputsrelating to orientation (e.g., direction). Orientation devices may alsobe co-located with a camera or built-in to the same overall device asthe camera. For instance, a smartphone is a type of AR device thatfrequently includes one or more cameras, a GPS unit, a gyroscope, and adigital compass built in to the smartphone.

Processors of exemplary embodiments executing predetermined computerinstructions are configured by the instructions to perform operationswhich control AR augmentations in dependence of the accuracy orconfidences measures which are obtained or obtainable from location andorientation sensors (e.g., the GPS unit, gyroscope, digital compass,etc.). The objective fulfilled by the processors executing augmentationselection or alteration algorithms is to generate augmentations (assensory output) that are selected or modified to convey accuracyinformation to a human user. An output of the processors is data that isultimately used by an output device such as a display, speaker, and/orhaptic device that outputs the selected or modified augmentation asvisual, auditory, audiovisual, and/or tactile output.

While embodiments of the invention may improve the accuracy ofdetermined location information by some method or subroutine, thecentral feature of displaying augmentations which reflect accuracyinformation stands apart from actually improving accuracy. Displayingaugmentations which represent accuracy or confidence information has itsown independent utility. Circumstances arise when accuracy simply cannotbe improved or is not cost or time effective to be improved. Under thesecircumstances there is still an advantage to convey information to theuser about the accuracy or confidence.

Referring now to FIG. 1, an exemplary method 100 is disclosed forproviding an augmented reality. At block 101, location information isobtained for a camera. Field of view and orientation information for thecamera may also be obtained at this step. Obtaining location or poseinformation may comprise receiving this information at a processor. Theobtaining step may further include actually generating some poseinformation with sensors, such as a GPS sensor, gyroscope, and/ordigital compass. The obtaining step may also include obtaining from oneor more databases location information describing the locations of realworld objects and/or virtual objects, at least some which correspondwith available augmentations. At block 101, accuracy or confidenceinformation for the pose and location information is also obtained.Accuracy or confidence intervals may be obtained by a processor fromlocation or orientation sensors. Accuracy or confidence intervals mayalso or alternatively be computed by the processor. At block 102, theprocessor selects or modifies an augmentation for output based on theaccuracy or confidence of the orientation and/or location information.For example, blur of an augmentation may be selected based on aconfidence interval (a high confidence results in the selection of acomparatively sharp augmentation image, and a low confidence results inthe selection of a comparatively blurry augmentation image). At block103, the augmentation is output with an augmented reality output device.

FIG. 2 is a diagram of an exemplary system 200 for performing the method100 of FIG. 1. The system 200 comprises an AR device 201 (e.g., asmartphone, tablet, special purpose AR headset, etc.) that includes oneor more cameras to capture images or videos. The AR device 201 may beconnected over a network to databases 205 comprising location andorientation accuracy information. Cloud computing devices 203 such asremote processors (e.g., server processors) may receive theimages/videos and the location information from the device 201 alongwith location and accuracy information from databases 205. Theprocessors then apply selection and/or modification filters to availableaugmentations based on the accuracy information to determine whataugmentations to present to a user. Augmentation (e.g., overlay) data isreturned to the device 201 which then generates a display 206 with theaugmentations. The augmentations may be one or more of the following:audio, visual, and/or tactile augmentation outputs. Processors of thedevice 201 may also or alternatively perform the data processing fordetermining the augmentations for output.

FIGS. 3A and 3B shows opposite sides of an exemplary AR device 300. FIG.3A shows the device 300 (e.g., a mobile phone) with a display 301 (e.g.,a screen). The electronic device 300 includes a speaker 302 and a hapticdevice 303 as additional output devices besides the display 301. Thedisplay 301, speaker 302, and haptic device 303 are all accuratelyregarded as “output devices”, as is the entire electronic device 300 byvirtue of the display, speaker, and haptic device being integraltherewith. The electronic device 300 comprises or is connected withadditional sensors such as an accelerometer 306, gyroscope 307, magneticfield sensor or magnetometer 308, proximity sensor 309, barometer 310,thermometer 311, and microphone 312. The sensors collect the respectivedata of their respective types (e.g., magnetometer collects magneticfield data or compass data).

Images or videos of a real world view of a geographic space are capturedusing one or more cameras. FIG. 3B shows a camera 304 on the rear sideof the electronic device 300. As used herein, “camera” is a devicecapable of capturing and characterizing incident electromagneticradiation (i.e., “light”) so as to recreate a visual image as aphotograph or a series of images forming a film or video. Cameras ofsome embodiments capture only the visible spectrum (i.e., what humanssee naturally). While general consumer cameras concern only the visualspectrum detectable by the unaided human eye, other embodiments of theinvention may use one or more cameras which are capable of capturingwavelengths of light which are not visible to unaided human eyes, forinstance infrared or ultraviolet light. The image (or images) capturedby the camera is characterized by data that describes both contents ofthe image (e.g., colors, pixels, etc.) and aspects of the image'scapture. The capture of an image is characterizable with pose (whichincludes both position and orientation) and field of view.

A real world image may include (e.g., if from a city's streetintersection camera for instance) HUD displays of date and time, or evencould have augmentations in it from another augmented reality systemthat is providing video to a system based on the present disclosure. Ingeneral an augmented reality system need only have some portion of itsinput that is real. In some embodiments this may be a relatively smallportion. Augmented reality systems may be used to modify theaugmentations of other augmented reality systems in more complexapplications, e.g., a system comprises distributed independentaugmentation engines which make use of each other's output.

The data from the camera(s) 304 and collected by the other sensors(e.g., 306, 307, 308, 309, 310, and/or 311) is received by one or moreprocessors 305. The camera data describes images or videos of a realworld view of the geographic space in the vicinity of the camera and, insome but not necessarily all embodiments, in the vicinity of theoperator of the camera. In this example, the camera 304 and the display301 are part of the same unitary electronic device 300, and thegeographic space is also in the vicinity of the output device, display301. The camera 304 and the electronic device 300 that includes thecamera 304 may be regarded as the viewing device. Viewing devices mayinclude various types (but not necessarily all types) of cameras, mobileelectronic devices, mobile phones, tablets, portable computers, wearabletechnology, and the like. If the electronic device 300 were ahead-mounted display (HMD), the HMD would be characterizable as aviewing device, too. A HMD that has no cameras, such as some see-throughHMDs, may still qualify as a viewing device. A lens or pair of lenses ofthe see-through head-mounted display also qualifies as a viewing device.

The one or more processors 305 are configured to process the data fromthe one or more cameras 304, as well as other data like data fromsensors 306, 307, 308, 309, 310, and/or 311, in order to generate anoutput useable by an output device to present an augmented reality to auser. In some embodiments, the image and/or sensor data from thecameras/sensors is sent over a network (e.g., the Internet) to one ormore remote servers comprising some of one or more processors thatperform processing of the data before augmentations are provided to anoutput device for outputting to a user. Such a networked implementationwas shown in FIG. 2.

An exemplary implementation of the invention may be in the context ofsafety systems. An AR system according to some exemplary embodiments maybe configured to warn users about dangerous areas where they should notwalk or travel, e.g., construction, sinkholes, rockslides, avalanches,and so on, by displaying augmentations within a live video streamdisplayed on a mobile device. An augmentation for this purpose may be,for example, a partly transparent color overlay that marks (e.g., bycolor such as red) the areas on the ground that are dangerous. Users maycheck at will for dangerous areas simply by looking at the ARapplication on their mobile devices and avoiding the problem areas,e.g., walking around them. A perfectly accurate system would simply markthe precise outlines of the danger areas. However, if in some situationsthe determination of the pose or field of view of the AR user becomesless certain, for safety reasons the augmentation boundary may beexpanded, so that the user would be in no danger even accounting forreduced location accuracy.

Another exemplary implementation provides improvements over indoor ARsystems such as the Microsoft HoloLens. Devices such as the HoloLens donot use GPS nor compass information, but instead uses multiple camerasand motion sensors to build an internal model of the indoor environment.The HoloLens then places augmentations into the internal model fordisplay to the user via a head mounted display. The HoloLens and similardevices may be used for a variety of purposes, recreational/consumer aswell as work/professional. According to an exemplary embodiment of theinvention, a physical device like the HoloLens may be configured with anAR software application that indicates the presence of water pipes orelectric power lines within walls, e.g., by marking the route of thosepipes or wires as an augmented reality line running along the wall.According to the exemplary embodiment, the output of the AR devicechanges based on the certainty/uncertainty of the positions ofstructures inside of the walls. For example, when the HoloLens or otherindoor AR device is relatively uncertain of its position, it mayaccordingly change the rendering of the augmentation. For example, theaugmentation may primarily consist of a solid line, but the line may berendered with increasing amounts of blur when positioning information isjudged to be less accurate. Conversely, with comparatively highpositioning accuracy, the line may be rendered with comparativelygreater sharpness or clarity.

Location information may be absolute (e.g., latitude, longitude,elevation, and a geodetic datum together may provide an absolutegeo-coded position requiring no additional information in order toidentify the location), relative (e.g., “2 blocks north of latitude30.39, longitude −97.71 provides position information relative to aseparately known absolute location), or associative (e.g., “right nextto the copy machine” provides location information if one already knowswhere the copy machine is; the location of the designated reference, inthis case the copy machine, may itself be absolute, relative, orassociative). Absolute location involving latitude and longitude may beassumed to include a standardized geodetic datum such as WGS84, theWorld Geodetic System 1984. In the United States and elsewhere thegeodetic datum is frequently ignored when discussing latitude andlongitude because the Global Positioning System (GPS) uses WGS84, andexpressions of latitude and longitude may be inherently assumed toinvolve this particular geodetic datum. For the present disclosure,absolute location information may use any suitable geodetic datum, WGS84or alternatives thereto.

It should be appreciated that “processor” is used herein in the singularfor easy and clarity of discussion. However, description of a step ormethod performed by “a processor” does not preclude one or moreprocessors from being involved in performing the described step ormethod. That is to say, “a processor” may operate independently or itmay operate with one or more additional processors to execute describedfunctionality. One or more processors may be located at differentgeographic places and networked together over the Internet, for example.

While the invention has been described herein in connection withexemplary embodiments and features, one skilled in the art willrecognize that the invention is not limited by the disclosure and thatvarious changes and modifications may be made without departing from thescope of the invention as defined by the appended claims.

What is claimed is:
 1. A method of providing an augmented reality,comprising obtaining location and orientation information for a camera;obtaining accuracy or confidence information for the location andorientation information; selecting or modifying an augmentation foroutput based on the determined accuracy or confidence; and outputtingthe augmentation with an augmented reality output device.
 2. The methodof claim 1, wherein the step of selecting or modifying comprisesselecting and modifying one or more of blur/sharpness, size/extent,color, and shape.
 3. The method of claim 2, wherein the step ofselecting or modifying comprises selecting a blurriness or sharpness ofan augmentation based on the accuracy or confidence.
 4. The method ofclaim 1, wherein the step of selecting or modifying comprises changingan uncertainty indicator that accompanies an unchanging base icon. 5.The method of claim 1, wherein the step of outputting comprisesoutputting at least one augmentation that is one or more of audial,visual, and tactile.
 6. The method of claim 1, wherein the step ofselecting or modifying outputs an augmentation which is a boundary, andwherein the step of selecting or modifying comprises expanding theboundary to account for a reduction in location accuracy.
 7. The methodof claim 1, wherein the step of selecting or modifying is based on thecertainty or uncertainty of the positions of structures inside of walls.8. The method of claim 1, wherein the first and second obtaining stepsand the selecting or modifying step are performed at least in part byone or more processors.
 9. The method of claim 8, wherein at least oneof the first and second obtaining steps is performed at least in part byone or more of a GPS unit, gyroscope, accelerometer, digital compass,and magnetometer.
 10. The method of claim 1, wherein the augmentedreality output device performing the outputting step is or comprises oneor more of a display, speaker, haptic device, smartphone, tablet, andspecial purpose AR headset.
 11. An augmented reality (AR) system,comprising a camera configured to capture images or videos; one or moreprocessors configured to execute computer readable instructions whichcause the one or more processors to perform obtaining location andorientation information for the camera, obtaining accuracy or confidenceinformation for the location and orientation information, and selectingor modifying an augmentation for output based on the determined accuracyor confidence; and an output device for outputting the augmentationselected or modified by the one or more processors.
 12. The AR system ofclaim 11, wherein the step of selecting or modifying comprises selectingand modifying one or more of blur/sharpness, size/extent, color, andshape.
 13. The AR system of claim 12, wherein the step of selecting ormodifying comprises selecting a blurriness or sharpness of anaugmentation based on the accuracy or confidence.
 14. The AR system ofclaim 11, wherein the step of selecting or modifying comprises changingan uncertainty indicator that accompanies an unchanging base icon. 15.The AR system of claim 11, wherein the output device is configured tooutput at least one augmentation that is one or more of audial, visual,and tactile.
 16. The AR system of claim 11, wherein the output device isconfigured to output an augmentation which is a boundary, and whereinthe step of selecting or modifying comprises expanding the boundary toaccount for a reduction in location accuracy.
 17. The AR system of claim11, wherein the step of selecting or modifying is based on the certaintyor uncertainty of the positions of structures inside of walls.
 18. TheAR system of claim 11, further comprising one or more of a GPS unit,gyroscope, accelerometer, digital compass, and magnetometer configuredto assist the one or more processors in performing the first and secondobtaining steps.
 19. The AR system of claim 11, wherein the outputdevice is or comprises one or more of a display, speaker, haptic device,smartphone, tablet, and special purpose AR headset.